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In June 1991, the MITRE Corporation submitted a series of recommendations as part of a Marshall 
Space Flight Center (MSFC) Management Information System Requirements Study, initiated by the 
Information Systems Office (ISO). A major recommendation of the study was to initiate the development 
of an Executive Information System (EIS) for MSFC executives. (2) ISO was directed, by center 
management, to proceed with the development of a Center-Wide Executive Information System. Existing 
EIS prototypes, developed by the Space Shuttle Projects Office and the Payload Projects Office, were 
reviewed. These existing MSFC prototypes were considered not to encompass the required functionality 
needed on a center-wide basis. A follow-up study by MITRE provided top-level system requirements. (4) 
These were later incorporated into a final requirements specification document by Boeing Computer 
Support Services. (1) 

Another MITRE study addressed the issue of whether to develop the Center-Wide EIS solely using 
in-house personnel and resources, or the purchase a Commercial Off-The-Shelf (COTS) product. (3). This 
second alternative was subsequently recommended and accepted by senior center management. A 
Request For Information (RFI), identifying system requirements, was then published in Commerce 
Business Daily. Vendor responses to the RFI were then reviewed. It was decided that the expertise of an 
"8A" contractor (designating a small, disadvantaged firm) would be utilized to evaluate the COTS 
products under consideration. The appropriate COTS product would then be purchased. The evaluation 
criteria, developed during this research, are intended to be used as a guide for software selection by the 
8A contractor. 

An Executive Information System is a computer system designed to support the informational 
needs of very senior executives. It is intended to provide timely, pertinent information to aid in decision- 
making, thereby eliminating the need to sift through lengthy reports. The concept of EIS is a relatively 
recent phenomenon. It was only in 1982 that Rockart and Treacy introduced the term to describe this 
emerging category of information systems. (7) An EIS may be defined as having the following 
characteristics: 

1. An easy to use and maintainable graphical user interface. 

2. Integrated capabilities for data access, security and control. 

3. On-request "drill-down" capability to lower levels of detail. 

4. Depiction of organizational health indicators. 

5. Functionality for decision support, ad-hoc queries and what-if analysis. 

6. Sophisticated tools for navigation. 

7. Data analysis features. 

8. Advanced report generation. 

9. Statistical analysis. 

10. Access to a variety of external data sources. 

EIS may be considered to be a subset of a broader group known as Management Information 
Systems (MIS). A variety of techniques have been developed to support evaluation of MIS proposals. 
Relevant published literature on this subject was reviewed, and various methodologies were investigated 
for their applicability. A weighted scoring technique was selected for the overall evaluation technique. It 
is considered to be a widely accepted technique by procurement agencies few evaluating "multiple 
proposals with varying prices and capabilities. (5) The scoring technique focuses on a listing of desired 
functional char acteristics Weights are assigned to each characteristic based on its perceived desirability. 
A composite score is then generated. The scoring technique was viewed as a means of fairly prioritizing 
MSFC requirements and quantifying the resulting vendor responses. It was also considered to be an 
effective means of countering any potential vendor protest. 

The eval uatio n process analyzes the vendor product alternatives from three perspectives: technical , 
cost and risk. The overall strategy for the evaluation is to eliminate any vendor alternative that cannot 
effectively meet the technical requirements. Cost and risk factors are considered only after all alternatives 
have been functionally reviewed. The composite scores for the technical, cost and risk analyses will be 
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factored and combined as identified in Equation 1, for any alternative that is deemed functionally 
acceptable: 

TOTAL SCORE = TECHNICAL EVALUATION (50%) + COST EVALUATION (30%) 

+ RISK EVALUATION (20%) 

The vendor alternative with the highest resulting score will then be purchased by the 8A contractor. 


The initial phase of the evaluation process utilizes three types of factors that are applied to each 
functional line item: requirements qualifier, weighting factor and scoring factor. Technical evaluation 
criteria were obtained primarily from the requirements specifications documents. (1, 4) E ach of the 
technical criteria have been qualified as to whether they are considered to be mandatory or highly 
desirable. The scoring factors vary with each designation. Each mandatory line item requires a specific 
response from the vendor. If a mandatory requirement is not bid by the vendor, it could be cause for 
disqualification. If a highly desirable requirement is not bid, it is not cause for disqualification, however a 
penalty is assessed via the scoring system for the absence of this functional capability. The weighting 
factor indicates the relative importance among the requirements. The weighting factors range from 1 
(low) to 10 (high). The scoring factor is used to rate each vendor’s response. The scoring factors are 
assigned relative to the following guidelines: 

Requirement Qualifier M H 

Have Capability Now +1 +1 

No Bid -3 -1 

The product of the scoring factor and the respective weighting factor for each line item is accumulated 
into the overall technical score for the vendor’s proposal. 

A series of forms were created in order to document the acceptability of the vendor product 
functionality, and to thus substantiate the overall technical evaluation. These consisted of a System Test 
Execution Log, Test Case Form, and a Problem Identification Form. The System Test Execution Log 
provides a summary of the individual test cases. The Test Case Form provides a standard format for 
documenting the original evaluation. For each line item requirement of the EIS Technical Evaluation 
Worksheet, a separate test case is conducted. Upon rejection of a test, a problem identification form is 
completed. The form provides documentation of system inadequacies. 

The cost analysis phase of the evaluation considers not only the price of the individual EIS software 
package itself, but of additional supporting hardware, annual licensing fees and operational transaction 
costs. These factors are combined in order to create a total systems cost. The Cost Analysis Summary 
Worksheet depicts the highest level of detail. Backup should be provided in order to indicate the pricing 
components of each major category. 

The final component of the evaluation involves assigning a confidence factor to each 
vendor/product. A risk assessment worksheet is used to factor each vendor's response to the RFI. The 
higher the rating , the higher the perceived credibility. Each vendor’s offering is evaluated based on the 
following general criteria: functionality, compatibility (with existing hardware, software and 
communications), installation, documentation, total systems cost, and vendor service. S imilar to the 
technical evaluation, the product of the weighting and scoring factors is used to establish a risk 
assessment score. 


Draft versions of analysis worksheets, test case forms and all criteria developed as part of this 
research, were submitted to personnel from the MSFC Information Systems Office and Boeing Computer 
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Support Services, for review and comment. Subsequent recommendations for modification were 
incorporated into the final report to the client. 


One of the major barriers to the success of Executive Information Systems has been a failure to 
consider post-implementation factors regarding these systems. For more traditional information systems 
applications (e.g. transactions processing, decision support systems), post-implementation issues have 
been recognized based on years of experience and much trial and error. Unfortunately this experience 
frequently can not be transferred directly to the EIS environment. The fragmented nature of executive 
work, the high degree of environmental uncertainty at this level in the organization, and the political 
ramifications of providing top management with more and better information, as well as other factors, 
make implementing (EIS) a special challenge." (6) The subsequent analysis focuses on some critical 
factors that should be considered following the implementation of the Center-Wide Executive 
Information System at MSFC. 


A post-implementation evaluation of the system should be conducted. This phase consists of two 
major components: the Development Recap and the Post-Implementation Review. The Development 
Recap provides an in-depth review of the EIS development activities that have just been completed. 
Analyses of cost and schedule variances are conducted as part of this recap. It should also identify 
describe and classify programming errors; suggest any needed revisions to the development methodology 
used; and provide any other relevant suggestions or insights. The Post-Implementation Review is 
performed four to six months after the system has been installed. The purpose of the review is to evaluate 
how well the EIS has performed in meeting the original expectations and projections. It is also intended 
to identify any further maintenance projects that should be undertaken to enhance or improve the 
implemented EIS. 

The value of any information system depends on the quality of its data; its timeliness, accessibility, 
accuracy and completeness. This is particularly true of Executive Information Systems. By attempting to 
provide high-quality data and information, an EIS may often highlight existing data management 
problems , while at the same time creating new ones. Analysis of the existing MSFC data environment 
iH<tnrififtii a series of issues regarding data availability, ownership, integrity, infrastructure, integration 
and management. Detailed recommendations were formulated regarding each of these areas. 

The p rimar y emphasis of the early post-implementation phase is to isolate and correct any system 
errors as soon as possible. A series of recommended procedures were suggested regarding this issue. The 
tami> overall process is used to address enhancements to the system. These enhancements should be 
compiled and evaluated after the primary fixes have been made. Because the executive user is 
particularly sensitive to changes in response time, the implications of any enhancement on this must be 
carefully considered. It is also important to emphasize that any changes involving the data sources must 
be co mmunicate d to the EIS maintenance group in order to avoid system malfunction. Current 
procedures regarding information system change control, configuration control, and version control were 
reviewed and found both adequate and applicable to the EIS. 

Consideration should be given to planning the post-implementation migration and evolution of the 
system. The EIS is likely to spread to additional users. The migration may be both hierarchical and 
lateral. With hierarchical migration, use of the EIS spreads from the top down in an organization. In 
lateral migration, use of the EIS moves across organizational units. The system is also likely to evolve to 
include information that is broader in scope, more detailed and closer to real time. The most effective 
method of EIS migration appears to utilize a strategic approach. The system is progressively made 
available to those executives where the need and expected return is the greatest. 

Construction of the Center-Wide EIS implies the need for a security system that will control access 
to sensitive information (e.g. Privacy Act information, long-range plans). In addition to prohibiting 
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access to the system by unauthorized personnel, the security system must be multi-level in nature, in 
order to be able to control access within a variety of classifications. According to the literature, there is a 
wide divergence of strategies in addressing this problem. Aerospace companies tend to use a multi-level 
security system utilizing the capabilities of the EIS application development software. An alternate 
approach utilizes a multi-level database to support data having different classifications and users having 
different clearances. Current plans for the MSFC EIS employ this last approach which is resident in the 
ORACLE source database. An ORACLE table will be used to identify user access privileges. This table 
will act in conjunction with queries associated with programmed SELECT statements on an individual 
field level. This planned security mechanism appears to satisfy the applicable NASA security 
requirements. Executives will want access to the EIS from remote sites. Security on dial-up access lines 
should be considered as a priority post-implementation system enhancement. The security system should 
also be as transparent as possible to the executive user. Multiple levels of passwords should be avoided. 
Most senior managers prefer "one-button" access to their systems. In the EIS this would be analogous to 
the WPS (Workstation Presentation Services) password synchronization feature that is currently used. 


In order to support the construction of an effective Executive Information System, this research has 
provided a quantitative method for assessing commercial off-the-shelf EIS development software. An 
analysis was also conducted which identified important post-implementation considerations. The final 
judgment concerning the effectiveness of the center-wide EIS will be determined lar gely by the executive 
user’s expectations and their perceptions of the system's ability to meet their particular support 
requirements. This highlights the need to obtain agreement from the user community on a clear set of 
metrics for determining system effectiveness, prior to actual system implementation. 
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